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ABSTRACT 


Telemedicine is a system of healthcare delivery tools which uses 
telecommunications consultations as an alternative to transportation of the patient. 
There are no conclusive studies to prove or disprove the use of telemedicine and it 
is often implemented with little basis for measuring its effectiveness. Recent 
initiatives have been driven by advances in technology and pressure by upper 
management to reduce the cost of health care, but not firom local needs 
assessments. This thesis provides a methodology to collect data used in supporting 
measures of effectiveness. The methodology is developed through a review of 
strategic goals, an assessment of potential measures of effectiveness, and use of a 
model for data collection. It is applied at a Navy medical treatment facility 
recently installing telemedicine equipment. 
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I. INTRODUCTION 


A. BACKGROUND 

Recently, mostly due to efforts to reduce health care costs, there has been much 

controversy over whether or not to investigate the effectiveness of telemedicine, and if so, 

how this should be done. Some believe that telemedicine is already on its way to 

acceptance and research is of secondary importance to the task of cultivating clinical 

telemedicine, while others believe that a systematic research agenda is important to the 

progression of telemedicine (Grigsby, 1996). In addition to the nation’s drive to reduce 

the cost of managed healthcare and improve outcomes, the Navy’s desire is to improve 

overall readiness and reduce movement of the patient. Part of the strategic goal 

formulated by the Navy’s Bureau of Medicine (BUMED) is to “encourage the use of 

appropriate available technologies to enhance and measure quality of care, readiness, 

patient satisfaction, resource utilization, and efficiency.” This has proven difficult, 

however, since it requires the collection and interpretation of data that may not be 

consistent across facilities. Furthermore, according to a January 1997 report to Congress 

by the Department of Commerce: 

“Although many individuals believe strongly in the potential of 
telemedicine for providing cost-effective services, not much "hard data" is 
available to support that belief. Decision-makers want to know the value- 
added of telemedicine. Lack of solid evaluative information is a significant 
barrier to the deployment of telemedicine.” 

There is not a clear methodology to collect data useful in making decisions about 
telemedicine. 

Outcomes research normally provides validation of effectiveness, but given the 
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variable nature of telemedicine, the study of telemedicine outcomes can be quite complex 
and, implementation can be costly in terms of manpower to analyze and make a 
comparison. This works against third-party payers who demand evidence that not only 
health services work, but that they are not excessively costly. 

In this thesis, I examine the deployment of a telemedicine suite during September 
1996. I identify a methodology for collecting data to establish measures of effectiveness, 
while discussing the use of a data collection model. I then focus on the U.S. Naval 
Hospital, Rota, Spain, as a case study for application of the model. The focus of this 
thesis is on using acquired information to decide where to place further investments in 
telemedicine. 

To understand the motives for this topic, reference is made to the DoD Guide for 

Managing Information Technology (IT) as an Investment and Measuring Performance 

which defines IT performance measurement as: 

“the assessment of effectiveness and efficiency of IT in support of the 
achievement of an organization’s missions, goals, and quantitative 
objectives through the application of outcome-based, measurable, and 
quantifiable criteria, compared against an established baseline, to 
activities, operations, and processes.” 

B. RESEARCH QUESTIONS 

Using the Naval Hospital Rota as a case study, I seek to answer the following questions: 

1. How can measurements be developed which support telemedicine 
effectiveness? 

2. Based on these measures of effectiveness, how can we use them to 
make better decisions on future investments of telemedicine to result in 
increased use? 

C. SCOPE AND LIMITATIONS 

The scope of this thesis includes a review of telemedicine, an analysis of current 
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efforts to measure the effectiveness of telemedicine, a proposed data collection 
methodology, and a case study of the deployment of telemedicine at Naval Hospital, 

Rota, Spain. Since this is a qualitative study of telemedicine functions, the technology 
discussions and statistical inferences will be limited and designed to address functional 
issues. 

D. METHODOLOGY 

The methodology used in this thesis includes a literature review, consultation with 

Navy telemedicine project personnel, analysis of proposed measures of effectiveness 

(MOE’s), a proposal for a data collection methodology, and a case study. The literature 

review consists of books, magazine articles, CD-ROM, the World Wide Web, and other 

library information services describing telemedicine applications and technology. 

Consultation was conducted with Chief Information Officer’s (CIO’s) and healthcare 

providers at Naval hospitals. A case study at the Naval Hospital Rota, Spain, was 

conducted to test the applicability of the MOEs proposed in this research. 

The case study focused on examining the deployment of telemedicine to a medical 

treatment facility with its own specific mission and requirements. It was not intended to 

use sophisticated statistical techniques in the analysis of survey responses, since the 

results were intended only to provide general guidance and not stringent findings. It also 

was not intended to infer that the methodologies used are ideal for other sites or 

requirements. As Robert Yin states in Case Study Research Design and Methods: 

Case study conclusions are generalizations to theoretical propositions and 
not to populations or universes.... In this sense a case study does not 
represent a ‘sample’ and the investigator’s goal is to expand and 
generalize theories (analytic generalization) and not to enumerate 
frequencies (statistical generalization). (Yin, 1988, p.21) 
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E. ORGANIZATION OF THESIS 


This thesis is composed of six chapters. Chapter II is a general review of 
telemedicine and existing barriers to implementation. Chapter III contains definitions 
and methods based on quality assurance and outcomes. Chapter IV proposes MOEs to be 
used in the case study and Chapter V uses a data collection model to apply these MOEs to 
the hospital in Rota. Chapter VI sxunmarizes the thesis and presents conclusions and 
recommendations drawn from the case study. 
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II. TELEMEDICINE BACKGROUND 


A. DEFINITION OF TELEMEDICINE 

Health Affairs last defined telemedicine in 1995: 

“Telemedicine is the investigation, monitoring, and management of 
patients and the education of patients and staff using systems which allow 
ready access to expert advice and patient information no matter where the 
patient or the relevant information is located.” 


Another definition firom “Telemedicine Today” (Allen & Perednia, 1996) is as follows: 

“Telemedicine is most commonly defined as the provision of medical care, 
services, and education through the use of telecommunications 
technology. The technologies used range fi:om the mundane (phones and 
FAX machines) to the fantastic (satellite-mediated virtual reality), and the 
medical applications range from transmission of medical records to remote 
telesurgery.” 


The GAO Report to Congress on Telemedicine (1997) describes it in three levels: 

“At the lowest level, telemedicine could be the exchange of health or 
medical information via the telephone or facsimile (fax) machine. At the 
next level, telemedicine could be the exchange of data and image 
information on a delayed basis. A third level could involve interactive 
audio-visual consultations between medical provider and patient using 
high-resolution monitors, cameras, and electronic stethoscopes.” 


Each definition has a consistent information technology and medical theme; however, a 
uni versally accepted definition is still a point of contention. It is this impreciseness that 
makes research and measurements more complex. 

1. Clinical Applications 

Most of today’s telemedicine practice falls into one of the following categories 
(Allen & Perednia, 1996): 
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• Interactive-video (lATV) mediated patient-physician consultations 

• lATV-mediated healthcare education 

• Teleradiology 

• Telepathology 

• Miscellaneous store-and-forward (S&F) applications: teledermatology, 

tele-ophthalmology, and transfer of medical information on the Internet. 

These functions are performed with the use of peripheral equipment such as video¬ 
teleconference (VTC) equipment, electronic stethoscopes, oto-ophthalmoscopes, and 
close-up cameras. 

The distinguishing feature of telemedicine is the communications link between 
devices. This link provides bandwidth, or a “pipeline,” and is the determining factor of 
how much data can be transmitted over a given time interval. The cost of bandwidth is 
also a varying factor depending on the circuit, the devices coimecting them, and the 
switches in between. Thus, although bandwidth is usually xmderstood to be more 
expensive as the capacity increases, sometimes it can be quite inexpensive. Generally the 
lATV applications require more bandwidth than other telemedicine applications. 

Applications which thrive are those which are not typically available by 
traditional means. Some medical applications are less sensitive to tactile requirements of 
healthcare and better suited for remote consultation through the use of 
telecommunications technology. Disciplines and applications that are weighted toward 
cognitive, visual, and aural information are more suited to telemedicine. Data, images, 
and speech are most faithfully transmitted in this way. The more dependent an 
interaction is on touch or smell, the less likely it is that telemedicine can fully substitute 
for an on-site interaction. Thus, according to Allen (1996), clinical applications of 
telemedicine today (from most to least used) are: 
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• Radiology (S&F; can be done over regular phone lines) 

• Psychiatry (lATV; has been used with ‘low-end’ systems running at 128 Kbps) 

• Dermatology (S&F; lATV) 

• Internal medicine specialties: (usually lATV) 

Cardiology 

Other (oncology, endocrinology, infections disease, etc.) 

• Neurology (lATV) 

• Home health care (lATV; over 50% of home nurse visits do not require hands- 

on interaction 

• Surgery (lATV) 

Follow-ups and initial assessments 

• Emergency room (lATV; links between small hospitals and urban E.R.s) 

• Nursing: (lATV) 

Counseling 

Education 

• Pathology (S&F; lATV) 

• Allied health: (lATV) 

Rehabilitation 
Physical therapy 

• Ophthalmology (S&F; lATV) 

• Dentistry (lATV) 


2. Telemedicine Technology 

In order to provide an operational telemedicine suite, there are three basic 
technical elements that must be in place and integrated: end components, standards, and 
comm uni cations links. 

a. End Components 

Components are being developed with increasing intensity in order to 
satisfy the demand of an increasing set of applications. The most commonly used 
instruments are digital stethoscopes, digital ECG’s, video otoscopes, surgical and patient 
cameras, digital ultrasound, dermatology microscopes, video ophthalmoscopes, and 
dental cameras. As new equipment is introduced, it must be evaluated as a medical 
device for use by telemedicine by the Food and Drug Administration (FDA). The 
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Medical Device Amendments of 1976 (Public Law 94-295) and the Safe Medical Device 
Amendments of 1990 (Public Law 101-629) established a comprehensive system for the 
regulation of medical devices intended for hmnan use. This can sometimes prove time- ■ 
consuming since lack of classification regulations has not allowed the FDA to establish 
exemption from any of the general controls for some of the devices where it would be 
appropriate. For example the use of a personal computer to view medical images could 
be classified as a "medical device" and subject to FDA approval. To remedy this 
situation a classification proposal has been developed and is awaiting Federal Register 
publication establishing classifications for five medical image management devices 
(FDA, 1996). 

b. Standards 

Standards are being sought for all aspects of telemedicine networks, but 
few have been implemented with the exception of the American College of Radiology 
Teleradiology Standard (United States Department of Commerce, 1997) which includes 
equipment guidelines. In order for a telemedicine system to work, the standards that 
must be in place are communications standards, file compression standards, and 
equipment standards. These standards offer a more reliable and robust link, and a 
standard applied consistently across multiple activities offers ensured coimectivity with 
more sites. Agencies with active roles in producing acceptable standards for medical 
equipment and telecommunications include the Food and Drug Administration (FDA), 
the Federal Communications Commission (FCC) Advisory Committee on 
Telecommunications and Health Care, and the National Institute of Standards and 
Technology (NIST). Within the FDA, the Center for Devices and Radiological Health 
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recognizes that the proliferation of “wireless” communications that can link the devices 
together and aid in the diagnosis and treatment of patients has introduced electromagnetic 
interference (EMI) with medical devices and has become a source of concern. 

Interference can cause problems with other communications devices or devices dependent 
on reception signals. Direct wired communications devices can suffer from EMI. The 
impact of these technologies must be addressed in the formation of standards (FDA, 
1996). 

c. Communications links 

Telecommunications is the most difficult technology to plan for and 
mamtain^ since telecommunications technologies are evolving at such a rapid rate. 
Equipment and protocols must be compatible and integratable to connectable sites. 
P lanning is further complicated by acquisition regulations, financial costs, and anticipated 
use of the link. Following are some of the technologies to be considered when planning 
telemedicine links. 

The Public Switched Telephone Network (PSTN) is the most commonly 
accessible. It is a “common carrier” and available to anyone or any organization who can 
afford to use it. Most of the services using PSTN are traditional voice, facsimile, or data 
transmissions. Since PSTN was originally designed for voice transmission, it is limited 
in speed and quality, thus high quality VTC and high resolution image files are not 
always possible. A 2400 baud rate is typical, but compression techniques can improve 
the rate to 56 Kbps. 

Another option is the Integrated Systems Digital Network (ISDN). In this 
network, all traffic is digitally encoded allowing use of more intelligent services using 
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compression and allowing greater capacity. The main advantage is that several services 
can be operated simultaneously from one line. For example, two researchers could be 
looking at separate views of the same file being transmitted, while talking on the phone 
or receiving a fax. Although the same PSTN line can be used, new switches must be 
installed to accommodate the digital requirement and thus may be expensive to 
implement. 

The Broadband Switched Network acts much like a multimedia 
workstation in which a physician receives many types of interactive portions of a consult. 
By definition, broadband offers higher than a 1.54 Mbps rate; however, the network 
would have to be rebuilt for switched broadband with new wiring, connections, and 
signaling algorithms. 

The Internet is the newest and fastest growing application of networking 
and is being integrated not only into medicine but also into everyday life. Some 
telemedicine applications are already being implemented on the Internet. The Internet 
grew out of a requirement by the Advanced Research Projects Agency (ARPA) for shared 
information among computers. Using the Transmission Control Protocol/Intemet 
Protocol (TCP/IP) it has evolved into a network of networks and an inexpensive way to 
send, receive, and view information in the form of text or graphics. It is now also 
possible to use interactive audio over the Internet. An example of the benefits of the 
Internet is a project with the South Texas Health Research Center (STHRC). STHRC 
originally desired to link all the hospitals, clinics, and training sites in South Texas in a 
sophisticated telecommunications network. This, however, would have proven to be cost 
prohibitive so they designed a shortcut to (1) create computer-stored files of the STHRC 
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inf ormation often requested out in the field, (2) install those files in an STHRC “file 
server” (essentially a central computer), (3) connect that STHRC server to the Internet 
with appropriate menus and file indexes, (4) encourage local clinics and other sites to link 
their desktop computers via local phone lines or state telecommunications lines to the 
nearest Internet access point, and (5) develop training so individual users in the field will 
know the few simple steps for connecting to STHRC via the Internet to request needed 
information or to exchange e-mail.” Even though bells and whistles were not 
implemented with the solution, the functionality of a telemedicine service exists, and as 
usage increases or decreases a more appropriate solution can be achieved incrementally. 

Another evolving link is through the use of digital radio. This provides 
greater portability of terminals,-services, and integration of broadcast and wired networks. 
Radio is not currently used as expansively as hard wire, although it could be a viable 
alternative in consultations occurring in extremely remote areas. 

B. fflSTORY 

Moore (1995) dates telemedicine back to the 1920s, when radio was used to link 
public health physicians standing watch at shore stations in order to assist ships at sea that 
had medical emergencies. However, one of the first organized developments of 
telemedicine was that by NASA in the early 1960s when humans began flying in space. 
Physiological parameters were monitored from both the spacecraft and the space suits 
during missions (Bashshur, 1975). 

One of the earliest deliveries of medical care with the use of telemedicine was 
also conceived by NASA. The Space Technology Applied to Rural Papago Advanced 
Health Care (STARPAHC) delivered medical care to the Papago Indian Reservation in 


11 



Arizona from 1972 to 1975 while also providing health care to astronauts in space. The 
instruments included an electrocardiograph and x-ray and were linked with a two-way 
microwave telemedicine and audio transmission (Bashshur, 1980). 

The Nebraska Psychiatric Institute was one of the first facilities in the country to 
have closed-circuit television m 1955. The link was with Norfolk State Hospital which 
was 112 miles away and was used for consultations, education, and later experimentation 
with group therapy (Benschoter, 1971). 

There have been numerous other miscellaneous applications of telemedicine. In 
1967, Massachusetts General Hospital and Logan International Airport Medical Station 
occupational health services were provided to airport employees and emergency care was 
provided to travelers (Murphy, 1973). In 1971,26 sites in Alaska were chosen by the 
National Library of Medicine to see if reliable communication would improve village 
health care (Foote, 1976). The first international telemedicine program was conducted by 
NASA in 1989 as a result of a massive earthquake in the Soviet Republic of Armenia. 
Under the auspices of the U.S./U.S.S.R. Joint Working Group on Space Biology, 
telemedicine consultations were conducted using one-way, voice, and facsimile between 
a medical center in Yerevan, Armenia and four medical centers in the U.S. This project 
demonstrated that medical consultation could be conducted over a satellite network 
crossing political, cultural, social, and economic borders. Even more recently, NASA’s 
Lewis Research Center in Cleveland, OH worked -with NASA’s Ames Research Center in 
Moffett Field, CA to provide echocardiographic examinations to be remotely diagnosed 
by a doctor at the Cleveland Clinic. It is hoped that this will pave the way for physicians 
on Earth to view the heart function of an astronaut aboard the International Space Station 
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(Nolan-Proxmire, 1997). 

Even though telemedicine has existed for decades, the official start of 
telemedicine in DoD is September 29, 1994, when the U.S. Army Medical Research and 
Materiel Command published a draft Telemedicine Testbed Master Plan. In January 
1995, it published a draft Telemedicine Strategic Plan. Since then, initiatives have been 
ongoing to include validation initiatives at the Naval Health Research Center (NHRC). 
Other leading DoD organizations involved in formulating national and DoD telemedicine 
policy initiatives include the DoD Advanced Research Projects Agency, the Office of the 
ASD(HA), and the U.S. Army Medical Research and Materiel Command. The ASD(HA) 
has established a DoD Telemedicine Testbed Board of Directors which supports national 
information infrastructure initiatives and is the steering group for DoD telemedicine 
development and utilization initiatives. However, even though collaboration has begim to 
establish a standard methodology for assessing the effectiveness of telemedicine, it has 
yet to be accepted and supported service-wide, and local sites are left with the task of 
setting strategic goals for the implementation of telemedicine. 

C. EXISTING BARRIERS 

Looking past the lack of project evaluation, there are still consistent and 
recognizable barriers to realizing the full benefits of telemedicine. GAO (1997) has 
acknowledged that these barriers can be categorized as legal and regulatory, financial,, 
technical, and cultural. They all have a basis for deterrent, and when one barrier is 
overcome, the other three can still cause a particular program to fail. The advantage for 
the Federal Government is that it is less affected by legal and regulatory barriers due to 
the Federal Tort Claims Act (FTCA). This is why most organizations are looking to the 
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Federal government to investigate new protocols and technologies and to conduct studies. 

1. Legal Barriers 

The most significant barrier in private telemedicine practice is the requirement for 
health care professionals to be credentialed in the state in which care is provided. Some 
states prohibit care provided through telemedicine (GAO, 1997). Although special 
licenses are available, they are not always accepted by other states. For federally 
employed professionals this is not an issue, since they must simply be licensed in one 
state in order to provide care to a patient eligible for federal benefits. However, this 
affects DoD healthcare if a civilian physician is prohibited by state law to practice 
telemedicine. There are a variety of models imder review that could be applied to a 
health care professional such as consulting exceptions, endorsement, mutual recognition, 
reciprocity, registration, limited licensure, national licensure, and federal licensure. 

These models are a long way from approval by all states, however. This is an area where 
the Federal government might serve as a model for states to follow. 

Malpractice is also an issue for every doctor-patient encounter. Currently a 
physician is held responsible for the treatment given to patients. However, if the consult 
is by telemedicine it is difficult to determine in which geographical jurisdiction the suit 
should be addressed. With federal employees, the controlling law is the Federal Tort 
Claims Act (FTCA), which for more than forty years “has been the legal mechanism for 
compensating persons injured by negligent or wrongful acts of Federal employees 
committed within the scope of their employment.” FTCA prevents federal employees 
from being, sued-rather, it is an action against the United States. As of the 1997 GAO 
report, only 4 known suits had been filed against private physicians, and no suits had 
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been filed against the federal government. 

The privacy and security of medical data must also be addressed. For example, 
those records that are considered classified (as in patients with AIDS) are at increasing 
risk when being transferred over public communications links. This is compounded by a 
failure for facilities to have policies and procedures in place to address cases that could be 
construed as sensitive. Within DoD, medical data is protected in accordance with the 
Privacy Act of 1974 and the DoD AIS Security Program which mandates the employment 
of technical and physical safeguards to ensure the information is adequately protected by 
health care providers and administrators. 

2. Financial Barriers 

Difficulties in determining reimbursements have deterred the expansion of 
telemedicine services. Often a consulting physician is not given monetary or workload 
credit for performing a tele-consult. In addition. Medicare does not pay for telemedicine 
because it believes the standard practice of medicine requires an “in-person, face-to-face 
consultation” between the patient and practitioner for those consultations requiring 
dialogue. However, in October 1996, HCFA announced that it will begin limited 
Medicare payments for telemedicine consultation in four states under a demonstration 
project (GAO, 1997). In a managed care organization increased access may result in 
increased utilization and thus increased cost. However, expanded use of capitated 
managed care systems should enhance the appeal of telemedicine and reduce the need for 
reimbursement. 

The financial barrier to smaller hospitals or clinics is the cost of implementing a 
telemedicine infrastructure which can range from several thousand to hundreds of 
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thousands of dollars. Even though the benefits of a high-bandwidth connection from a 
small rural hospital to a major health care center are high, smaller facilities may be the 
least able to pay for these services. In these cases, resource planners must be able to 
leverage additional services with the infrastructure installation, such as education or 
entertainment. 

3. Technical Barriers 

Technical standards are slow to be adopted, such as data definition, coding, and 
transmission of images. These distinctions are very important to the payer and provider, 
since they can affect which insurance company will be liable for a claim. Much of the 
information is left to the interpretation of individual managed care organizations, and 
providers must also make assumptions when coding claim data elements and frequently 
use coding standards employed by the provider’s system. To date, the only specialty¬ 
generated clinical guidelines existing are the practice guidelines for teleradiology 
developed by the American College of Radiology (United States Department of 
Commerce, 1997). 

Lack of technical standards can result in equipment and systems that do not 
communicate with other equipment and systems or does not provide adequate images for 
clinical decision-making. All too often, lack of objective technical advice has led to 
inappropriate purchases and poorly performing systems that hinder the cost-effective , 
application of telemedicine technologies. Equipment is still installed with proprietary 
and “closed” standards in which the equipment may only communicate with other suites 
of its kind. This is especially a problem in DoD, where portability and movement is 
always an issue—the location of the patient or the needed specialty could be anywhere in 
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the world. 


4. Cultural Barriers 

Cultural acceptance usually falls into two categories: physician acceptance (to • 
include willingness to use the equipment and their skepticism about diagnosing and 
treating patients without a face-to-face encounter) and patient satisfaction. 

Use by conservative physicians is dependent upon their desire to provide quality, 
controlled, convenient, and low-risk care to the patient. Given the lack of telemedicine 
research, there is a substantial level of resistance to accept the process of telemedicine 
consultation. Convincing physicians that telemedicine can provide appropriate, if not 
better, level of care requires significant training, discussion with peers, and 
experimentation by the physician. Downplay of physician participation has been 
documented such as in a 1995 journal article by the Health and Human Services (HHS) 
and the Telemedicine Center, Medical College of Georgia, in which it is stated that the 
designs of current systems are driven more by technology than by the needs of physicians 
(Puskin and Sanders, 1995). In these cases, telemedicine technologies must adapt to the 
needs of users, not vice versa, in order to optimize the buy-in of healthcare providers. 
Training was cited by Puskin and Sanders as a key component of any successful 
telemedicine system. 

Patient satisfaction, according to GAO, may be less of a barrier than physician 
acceptance, particularly in rural settings. As compared with driving hundreds of miles for 
health care, telemedicine is a much more appealing option than seeing a specialist face- 
to-face. Projects in South Dakota and Florida formd approximately 70 percent of the 
patients preferred telemedicine consultations and 14 percent were neutral. Another 
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project in Kansas involved patients who were content avoiding a 300 or 400 mile drive; 
however, they found it difficult to be candid on video and were not eager to repeat their 
experiences (GAO, 1997). In DoD, however, it is difficult to measure patient satisfaction 
in an overseas setting, since it is often a benefit in the patient’s eyes to be transported to a 
major hospital for the purpose of improved morale. Patients are more “satisfied” if the 
location of treatment is “home.” This is where cultural differences surface regarding 
civilian and military settings which affect the perception of telemedicine. 
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III. MEASURES OF EFFECTIVENESS AND BACKGROUND 
A. DEFINITIONS 

There are many different ways to measure effectiveness of clinical information 
systems. The purpose of measures of effectiveness is to discern the extent to which an 
objective is achieved. The Joint Medical Operations-Telemedicine Advanced Concepts 
Technology Demonstration (JMO-T ACTD) defines an MOE as a "quantifiable measure 
used in comparing systems or concepts or estimating the contribution of a system or 
concept to the effectiveness of a military force. It expresses the extent to which a combat 
system accomplishes or supports a military mission." There are also measures of 
performance which help assess effectiveness; these are usually more quantifiable points 
such as response time, exceptions, and errors derived from a piece of equipment or an 
information system. The JMO-T ACTD defines a measure of performance (MOP) as an 
MOE "that expresses the extent to which a combat system accomplishes a specific 
performance function. In general, higher-level MOPs are themselves composed of either 
lower-level MOPs or data requirements." A system can be considered effective if it 
meets its performance requirements or it contributes in a significant way to fulfilling long 
term strategic needs within an allocated budget (Whatmore, 1995). Measmes of 
performance are concerned with whether a system is “doing it right.” Measures of 
effectiveness, however, are concerned with “doing the right thing.” Thus, if an objective 
is for a system to provide optimal readiness, measures of effectiveness should measure 
achievement of that objective. 

Descriptions of effectiveness measures in the literature are typically conflicting or 
vague. There are many definitions of efficiency, effectiveness, cost/benefit analyses. 
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performance measurements, goals, outputs, and objectives and the terms are sometimes 
used interchangeably. It is also common to find generalizations with little tangible 
support. Healthcare is a challenge for managers wishing to collect measures of 
effectiveness since the objectives are difficult to define, as is the definition of the quality 
of health care. The health care industry is laden with intangible costs and benefits, and it 
is this ambiguity that makes data difficult to gather and interpret. It is not unusual or 
necessarily inappropriate to find disagreement in deciding what to measure. It has been 
noted (Connelly, 1993) that the method for generating measures of effectiveness rests on 
the principal that an MOE is “acceptable to an individual only if it rates effectiveness in a 
manner consistent tvith the ratings of that individual whose judgments are important for 
the area of interest.” 

B. METHODS FOR DEFINING MOE’S 

In research, measurement consists of assigning numbers to empirical events in 
compliance with a set of rules (Cooper & Emory, 1995), but it is not easy to measure 
properties like satisfaction-one must infer its presence or absence by observing some 
indicant or pointer measurement. Nonetheless, measures must start somewhere and 
objectives, or goals that are being sought or maintained, are the foundation. 
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Strategic Effectiveness 
Mission Effectiveness 

System Effectiveness 

A I 

Equipment Effectiveness 

^ Equipment Performance 

Figure 1. Hierarchy of Objectives and Effectiveness 
(Whatmore, 1995) 

There is a hierarchy of objectives which naturally evolve into a corresponding 
hierarchy of effectiveness as shown in Figure 1 (Whatmore, 1995). Objectives define the 
requirements for lower levels until a performance measure is identified to support the 
measurement of effectiveness of the objectives. This is a good illustration of the process 
that occurs within the monitoring of systems. It also helps in isolating separate factors 
for integrated systems in which, for one strategic objective, there are several mission 
objectives. 

Another approach published by Wakulc 2 yk describes a process that he states is a 
low-risk, methodic approach to selecting metrics for a metrics program. He states that 
there is a lack of software managers of practitioners who have experience applying 
influential references (most specifically the Malcolm Baldridge Award criteria and the 
Software Engineering Institute Capability Maturity Model for software). This implies 
that some organizations are inadvertently creating metrics programs without the required 
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foundation of organizational definition. Wakulczyk’s method describes a three- 
component approach that considers organizational definition to be a prerequisite to truly 
successful metrics: analyze the present, envision the future, THEN create metrics that 
will allow the organization to become its envisioned state. This is similar to the method 
illustrated by Whatmore, who attempts to identify objectives (strategic, mission, 
equipment) before identifying measures of effectiveness. It is the attempted use of these 
approaches by analyzing the present and future states and identifying the most apparent 
and visible measures based on the convergence of relevant data that is available. 

Once objectives have been identified, associated measures must be identified and 
defined, keeping in mind that a good measurement tool must be easy to use, and meet the 
criteria of validity, reliability, and practicality: 

“Validity refers to the extent to which a test measures what we 
actually wish to measure. Reliability has to do with the accuracy and 
precision of a measurement procedure... Practicality is concerned with a 
wide range of factors of economy, convenience, and interpretability.” 

(Thorndike & Hagen, 1977) 

C. QUALITY ASSURANCE AND OUTCOME MEASUREMENTS 

A common mistake is that an effort to identify appropriate measures exceeds the 
ability of the manager to collect the data. The data is then incorrectly manipulated imtil it 
is perceived that validity, reliability, or practicality has been met. 

1. Quality Assurance through Statistical Data 

The main vehicle by which health care is monitored is through a quality assurance 
program (sometimes called performance improvement). A good quality assurance 
program requires valid, reliable data. As defined by Sink (1985), quality is “the degree to 
which the system conforms to requirements, specifications, or expectations.” Quality 
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assurance exists in health care to assure beneficiaries that they are getting the service they 
are entitled and expect to receive. Currently, the Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO) is the leading organization responsible for 
conducting accreditation. According to standards set by JCAHO in 1995, health care 
organizations are expected to use aggregate data and information to examine trends over 
time, make comparisons within the organization and among organizations, and utilize 
management information obtained to improve performance. Quality assurance programs 
use a variety of benchmarks and standards (such as inpatient length of stay) which are 
typically published by JCAHO or BUMED, but individual sites are given latitude to 
implement their own overall program. 

2. Outcome Measurements 

Measuring outcomes represents a new strategic direction for managing health care 
experiencew for beneficiaries. It presents an opportunity to enhance patient care and 
decrease health care costs through stages of identification, intervention, and monitoring 
and evaluation. If collected and used appropriately, outcome data can be used to improve 
clinical efficiency and quality and feed that information back into a quality assurance and 
quality improvement programs to improve patients’ health care. Furthermore, outcomes 
can identify areas which require attention from senior managers (Kretz, 1996). Simply 
stated, if outcomes can be identified with a new process, a change in outcomes will affect 
decisions to alter or enhance the process. 

A pl ann ed change always results in resistance, thus a new way of practicing 
medicine results in careful scrutiny of changes in outcomes. Providers are under intense 
pressure to improve the quality of care, and introducing telemedicine has introduced risk 
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to the consistency and improvement of care. However, as an impetus to change, 

Congress is increasing pressure to evaluate telemedicine programs and to identify 
coordination among services. The Joint Working Group on Telemedicine (JWGT) has 
constructed a Federal evaluation framework made up of six areas, one of which is clinical 
outcomes. The question posed by GAO is, “Are acceptable outcomes associated with the 
use of telemedicine?” and hospitals must be prepared to justify the answer. GAO, 
however, has acknowledged that JWGT efforts at coordination among services have been 
hampered by several factors including lack of a consistent definition and incompatible 
agency data. A definition of outcomes from Avedis (1989) enforces this: “Outcomes are 
those changes, either favorable or adverse, in the actual or potential health status of 
persons, groups, or communities, that can be attributed to prior or current care.” Given 
this definition, few organizations are ready to deliver outcomes for 3 reasons, according 
to Kretz: 

1. The industry is segmented among local providers of care, mid-level 
companies that specialize in delivering one service, and large national 
companies. 

2. Outcomes analysts find it difficult and in some cases impossible to 
distinguish outcomes resulting from care decisions made by physicians 
from those made by home care providers because home care providers 
operate under the orders of the prescribing physician. 

3. The largest home care companies have the staff and financial resources 
needed to develop outcomes systems but face significant problems 
integrating their data systems so that they can collect and report outcomes 
data. 

The concept of outcomes-based quality improvement is simple to understand, but 
its implementation is not. The basic idea is to measure patients’ clinical status at two or 
more time points while keeping track of the kinds and intensity of care they receive. The 


24 



next step is to analyze the data to determine which processes are associated with good 
outcomes, which are associated with bad outcomes and modify practices (if necessary) in 
light of the results. In the context of clinical or health services research, an outcome is 
the difference between ratings during or following treatment. The effect of the treatment 
is the difference between the actual outcome and the expected outcome given no 
treatment. The problems associated with this method of outcome-based measurements are 
best illustrated by Grigsby (1996); 

“One problem is that all illnesses do not have the same course.... 

From one disease to another there is significant variability in what is 
considered favorable: stabilization, improvement, or slowing of 
progression may be good outcomes, depending on the disorder. Likewise, 
the metric may be quite different: improvement in activities of daily 
living, decrease in chronic pain, or resolution of dermatologic symptoms. 

One solution may involve stratification of patients into categories. You 
might examine all diabetics, or all Type II diabetics, or all persons with 
progressive neurodegenerative diseases, or persons with fractures of the 
lower extremities. Depending on your focus, outcome measures may be 
disease-specific, or global.” 

In short, Grigsby clarifies what is obvious to health care providers, but harder to see from 
the information systems manager’s point of view-that in order to evaluate outcomes, you 
have to remain open to the variations in disorders. In other words, when evaluating an 
outcome of a dermatology case versus an obstetrics case, “it all depends.” He goes on 
further to look at specific cases within a condition. 

“A second difficulty is that not all patients with a given condition 
are alike. This is where risk-adjustment becomes important. Assume you 
are a geriatrician, and you want to study the outcomes of telemedicine care 
for a sample of geriatric outpatients. Older persons with serious illnesses 
are an alphabet soup of comorbid conditions are liable to have worse 
outcomes, regardless of the care they receive, than are healthier persons. It 
may be that they also are more likely to receive consultative services via 
telemedicine because of the complexity of their condition. If they are 
compared, without some kind of risk-adjustment (by stratification or 
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statistical means), with individuals who are less ill and/or have a lower 
burden of comorbid illness, you may incorrectly demonstrate that 
telemedicine outcomes are worse than those of conventional care.” 

Morrissey (1997) summarizes another view, that outcomes can be measured according to 

clinical results, patient satisfaction, difference in functional ability and the costs of doing 

a procedure, but “process is the dial that you can adjust to affect the other four. If you 

improve the process, everything else will get better.” This idea appears to support the 

notion that although possible, outcomes is of secondary importance and can tend to be 

reactive rather than proactive. It does not solve the problem that most sites are facing- 

they can’t provide evidence that telemedicine is making things better. This is why 

measures of effectiveness are necessary. 
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IV. CRITICAL ASSESSMENT OF POTENTIAL TELEMEDICINE MOE’S 


“With the exception of image-oriented subspecialties, such as teleradiology and 
telepathology, few clinical studies have documented the accuracy, reliability, or clinical 
utility of most applications of telemedicine as a primary diagnostic or therapeutic 
modality” (Perednia, 1995). In general, most projects and programs have simply not 
generated enough telemedicine consultations by separate clinical procedures to make 
definite conclusions about the effectiveness of telemedicine, typically experiencing one 
of the previously discussed barriers that prevent it firom growing. There are random 
studies, but they have yet to produce accepted measures. It is rarely disputed that the 
technology works, but the effectiveness can be disputed; the problems lie in answering 
questions concerning the disposition of the technology. 

A. STEPS TO IDENTIFY MOE'S 

Telemedicine is a system composed of many different stimuli which are not 
always tangible and often result in integrated performance measures. So, to clarify, DoD 
seeks to manage information technology as an investment (with emphasis upon the 
methodology of selection, control, and evaluation) and to be objective and outcome- 
oriented. Two objectives of Navy Medicine is to reduce the number of medical 
evacuations and to optimize the time spent by specialists giving care (Greenauer, 1995). 
In addition, “Goal 3: Technology” of the BUMED Strategic Plan states that the “Navy 
Medical Department will be a leader in technology integration.” Strategies that support 
this goal are: 

• Pursue the development and deployment of digital management information 
systems. 
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• Ensure that Navy Medical communication systems are integrated to store and 
transfer medical information throughout the Department of Defense 

• Encourage the use of appropriate available technologies to enhance and 
measure quality of care, readiness, patient satisfaction, resource utilization, and 
efficiency. 

• Obtain and distribute current hardware and software and train Medical 
Department personnel for collection of accurate and timely information. 


The most applicable strategy to this thesis is the third, which defines the objectives of 
new technologies and identifies five factors to be used in creation of metrics. Although it 
is left up to local organizations to develop systems of measuring progress toward these 
goals, six measures are identified that are linked to one or more of these objectives as 
follows. (See Figure 2) 



Figure 2. Steps to identify MOEs 


B. MOE IDENTIFICATION 

Figure 3 shows the six proposed MOEs that when used together, can satisfy all the 
strategic goals defined in section IV.A. 
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Strategic Goal 
Quality of Care 


MOEs 

Outcomes 


Return to Duty Times 

Number of Medevacs 

Patient Satisfaction 
Surveys 

Physician Satisfaction 
Survey 

Utilization Rate 
Figure 3. Cross-Mapping of MOEs to Strategic Goals 

1. Outcome Measures (quality of care) 

The objective in outcome management is to optimize quality. As discussed 
previously, quality is a foundation for acceptance and adoption of new clinical processes. 
There is currently no standard tool in place to measure changes in health care outcomes 
due to telemedicine. Furthermore, personnel in quality assurance and/or performance 
improvement positions have no vehicle to identify whether the outcome of a telemedicine 
consult is different when telemedicine is used. Although some sites are formulating 
protocols to be used for telemedicine consults, none have been identified that consistently 
and primarily address outcomes. Outcomes measurement can be done however, on an 
audit basis through the use of a short form that directly asks the physician if the 
telemedicine consult has resulted in a a change of diagnosis, disposition, or treatment. If 
negative outcomes cannot be associated with the use of telemedicine (or positive 
outcomes can) it suggests that telemedicine is being effective. 
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2. Return to Duty Times (readiness) 

In DoD, the time span between time of injury and return to full duty is of 
particular interest, since the time spent by an active duty member receiving healthcare is 
time lost in performing his/her primary duties. In other words, the member is not 
available and readiness is reduced. Managers are interested in maximizing the time that a 
member is in full duty status. Thus, the benefit of telemedicine can be measured in terms 
of decreased time that it takes to travel to the treatment facility, receive treatment, and 
return. Depending on the duty location and the location of the medical facility, this could 
result in several hours or several weeks. Also, depending on the rank and duties of the 
patient, the manpower and monetary costs could vary significantly. In a thesis done by 
Craigmiles (1995), the travel time of a typical consult was 2 hours for a round trip and the 
typical patient was an E-1 in training. However, the average number of consults per year 
is 20,000 which results in a potential training time savings of40,000 hours. In contrast, 
in Bosnia, the round trip travel time was weeks, typically resulting in a medical 
evacuation to Germany, and the typical patient was a higher enlisted rate providing 
operational expertise. The number of consults was low, at 39 over two and a half months. 
Using an average of 2 weeks lost for each medevac, this results in a total time savings of 
13,104 hours. In both cases, this implies that readiness was improved since the patient 
was returned to full duty status quickly. Return to duty times were improved but 
comparing these times as discussed earlier can be misleading, since the definition of 
readiness is different in a variety of military settings and has yet to be defined with a 
standard set of indicators. Studies continue to use the pay grade of the service member to 
calculate a costs savings, but this is exceptionally subjective by the rater and nearly 
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impossible to standardize. What is more certain is the fact that successful telemedicine 
consults occurred which resulted in no need for travel. 

On the surface, return to duty times could be a useful measurement to contribute- 
to the level of readiness, however the validity is lost in the analysis of various scenarios 
that could prove to discount the measure. Specifically, telemedicine may be a reason for 
increasing the time before return to duty. Reliability, defined as the degree to which the 
measurement is free of error, is considered a necessary but insufficient condition for 
validity (Benbasat & Moore, 1992). In the case of return to duty times, it is difficult to 
estimate the level of error and whether the MOE would be overestimated or 
underestimated. The types of errors possible are illustrated in scenarios such as persons 
on limited duty but still being capable of performing their primary jobs. Furthermore, by 
identifying a condition, telemedicine can create a limited duty and prevent a casualty or 
medical retirement. These situations are easily lost and therefore not reported, or only 
partially reported. Thus effectiveness of telemedicine can not be determined by 
measuring return to duty times. 

3. Number of Medical Evacuations (Medevacs) (readiness) 

The largest and most complicated medical transportation system is operated by 
the DoD Military Health Services System. Economies of scale have forced the DoD to 
consolidate and specialize its hospitals and clinics. Consequently, specialized medical 
care cannot be provided at every in stallation. With personnel located at many, widely- 
dispersed DoD installations, the DoD is forced to transport a patient to the proper facility. 
It is proposed that telemedicine offers highly increased readiness and significant savings 
by preventing a number of medevacs. When a patient is in a remote area and the 
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attending physician does not have the resources to make an adequate diagnosis, the 
correct disposition defaults to a medevac. Of course, medevacs are expensive due to the 
cost of the flight, TAD costs, and the cost of losing the worker. Medevacs also include 
risks if the patient is in a combat zone or at sea. When patients are at sea, they must be 
transported by helicopter, or by “high-lining” patients between ships, where a cable is 
literally strung between the ships and, using pulleys, the patient is pulled to the other 
vessel. The strategic mission is delayed until this procedure has been completed and the 
member is on his way to safety. In its most simple form, telemedicine must be assumed 
to be effective if it causes the number of medevacs to be reduced. 

4. Patient Satisfaction Measurements (patient satisfaction) 

An easier measurement .that can be standardized in a one-size-fits-all manner, is a 
patient satisfaction survey, but results already published suggest that most patients find 
telemedicine an acceptable way to receive medical care (Grigsby, 1996). In a thesis done 
by Craigmiles (1995), satisfaction measurements of 52 patients within one month were 
measured and it was found that the telemedicine system neither positively nor negatively 
impacted the patients’ perception of staff attitude, the time spent in discussing the 
patients’ medical problems, or the overall perception of care. This is consistent with 
Grigsby’s theory that telemedicine has been generally accepted in a traditional setting. 

Although patient satisfaction remains a principal approach to identifying problems 
within the overall health care system, it loses its validity concerning telemedicine due to 
biases incurred in the attractiveness of a patient embarking on an incentive trip. Other 
factors can discredit patient satisfaction surveys, especially in the military where patients 
are stationed in remote areas and would be better suited to receive health care in a more 
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populated area. Moreover, patients typically desire movement to a more populated area. 
Patients must be allowed to provide feedback into the system by assessing their 
satisfaction with the entire process. However, if they know that this process will prevent 
them from being moved, results may be skewed since there are personal benefits to be 
realized by a medevac aside from the military mission. Morale is extremely important. 

In remote areas, members are typically under extreme pressure to perform imder stressful 
conditions. Liberty and leave are granted under the military’s purview and an 
opportunity for medical liberty or convalescent leave is attractive and sometimes the 
commanding officer will opt for the medical evacuation to take advantage of the possible 
rejuvenation of morale. This factor has been suggested as attributable to minimal use of 
telemedicine in Bosnia, since commanding officers do not want to be dealing with 
possible long-term care by either telemedicine or traditional care. Commanding Officers 
believe that the best place to receive care is outside of a stressful environment, and that 
the the member should be evacuated. Thus, there is a bias in the patient survey construct 
and it should only be used in stable environments, not for operational environments. 

5. Physician Satisfaction Measurements (efficiency, resource utilization) 

The objective in this factor is to provide a high technology environment as desired 
and/or needed by health care providers. The focus of many technologies is to improve 
efficiency and resource utilization-it does what it is supposed to do and it is being used. 

It is hardly sxirprising that satisfaction of the healthcare provider is often overlooked. 
Often the hype of technology or patient dissatisfaction encompasses telemedicine projects 
and the attitudes of the health care provider is not assessed. So when the equipment is 
not used, the information systems analyst is puzzled. It is true that addressing and 
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resolving technical issues is a vital part of developing a successful telemedicine progr am 
However, debates over what is adequate resolution quality, compression ratios, whose 
codec is better, what is adequate bandwidth, and the plethora of other technical issues 
represent in part a denial on the part of industry experts to grapple with the truly thorny 
issues facing telemedicine. It needs to be identified why or why not physicians choose 
telemedicine. An attitudes assessment will identify many issues that are not apparent to 
management personnel and provide the manager with benchmarks for improvement. 
Generally, if the health care providers accept and express satisfaction with what 
telemedicine is doing and express their intentions to use telemedicine, it can be assumed 
that there is a favorable impact on the effectiveness of telemedicine. 

6. Utilization Rate (resource utilization) 

There is an objective to decrease the time required of specialists (Greenauer, 
1995) and optimize the time giving care by specialists. This is a measure of the level at 
which telemedicine consults compare with traditional consults. By accounting for the 
number of telemedicine consults performed and received, it can be compared to the 
number of traditional consults in the same specialty. This number can also be compared 
with the survey data asking the health care provider what he anticipates his workload to 
be. In this way, there is a goal to be achieved, and although the goal may change, at least 
there is a method for knowing how we are moving toward or from the goal. The 
utilization rate is certainly not a measure of how well the equipment is working. It is a 
reflection of the performance aggregate spanning the entire spectrum of care, affected by 
other influences such as management practices, clinical protocols, education, and 
dialogue necessary to accomplish the mission of the facility. If telemedicine is being 
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used consistently or increasingly, it can lead to the understanding that telemedicine is 
being effective. 

C. PROPOSED SET OF MOE’S 

In summary, it is recommended that physician satisfaction measurements, 
utilization rates, outcomes measures, and number of medevacs be used as the foimdations 
for measures of effectiveness. Although this thesis does not go in depth to analyze their 
validity, their merits initially warrant use in this study. Next it becomes desirable to 
collect the data needed for these MOE’s in a practical manner. 
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V. CASE STUDY: IMPLEMENTATION OF TELEMEDICINE AT ROTA 

NAVAL HOSPITAL, SPAIN 

A. BACKGROUND 

The primary role of the Naval Station, Rota, Spain is to support the U.S. Sixth 
Fleet operating in the Mediterranean. While the Sixth Fleet tackles the increasingly 
complex task of promoting peace and stability in the Mediterranean area, the Naval 
Station Rota and its tenant commands maintain support to the operating forces. Naval 
Hospital, Rota is a forward deployed Medical Treatment Facility serving military 
members and their family members, retired members and their family members, and 
DOD contracted civilians stationed in Rota Spain. Areas of response include the Iberian 
Peninsula, Western Mediterranean, and North Africa while also providing support to the 
Sixth Fleet, DOD and NASA shore activities, and primary surgical team response for the 
European theater. Other hospitals in the area are Naval Hospitals Naples (with satellite 
clinics at Gaeta, LaMaddelena, Pintamare, and Capodichino) and Sigonella (with a clinic 
at Souda Bay). 

The head of the Management Information Systems Department (MISD) at Rota 
has been present for the full sequence of events of the telemedicine deployment at Rota 
and has been accepted by the command as the primary and credible source of information 
on use of telemedicine. He has remained proactive and informed on the progress of 
technology outside of Rota and has managed resources to remain integrated with DoD. 
During a site visit in April 1997, it was routinely asked of healthcare providers, “What 
would you recommend to be the next step in the use of telemedicine at Rota? Where 
would you invest resources?” The typical answer was “Fm not sure. I’d go to MISD.” 
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There are two clinical champions primarily responsible for encouraging clinical use. One 
is the flight surgeon, a computer “superuser” who encourages healthcare providers to look 
outside of Rota for a network of solutions. The other is the dermatologist, the primary ■ 
user, who is continuing to realize the benefits of off-site consultation and sees the 
advantages of using telemedicine to avoid moving the patient. Another advocate is the 
Commanding Officer, a neurosurgeon dedicated to encouraging use of telemedicine as he 
markets its successes and potential uses. His views on telemedicine are known 
throughout the command and most employees state that the successes at Rota are largely 
due to his support of the program. He does not believe that this new process warrants 
extra resources, however, and has resisted committing extra resources at this time. He 
emphasizes that telemedicine is a change in current processes and should not be confused 
with a new process. In summary. Rota is populated with clinical champions, an 
information systems advocate, and a central authority figure all in tune with driving the 
success of telemedicine. 

The Naval Medical Information Management Center (NMIMC) took the lead and 
deployed telemedicine to the European theater in September, 1996, funding the 
implementation costs. Included in the deployment was delivery of dermatology scopes, 
ENT scopes, and VTC units. There was no local plan or needs assessment prior to 
delivery and installation of the equipment; however, the sites were given ownership and 
entire control of the program. The deployment consisted of a site visit by NMIMC 
persoimel who configured and tested the equipment and trained local personnel. While 
the equipment was installed, NMIMC personnel held a two-day conference with key local 
players to discuss issues as follows: 
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• Customer expectations 

• Role of NMIMC Detachment in supporting existing and new infrastructure 

• Working knowledge of the effect of telemedicine and other technology on 
overseas sites 

• Ambulatory care data collection and business process reengineering efforts 

• Reemphasis to the COs and CIOs the significance of the LCM and funding 
changes as part of a long term strategic approach 

Collection of metrics was discussed, and although it was agreed that it was important and 
necessary, there was not a consensus on what metrics to gather. At a later conference, 
outcomes measures was committed as the highest priority of measurements. There had 
been no prior metrics gathered to use as a comparison and a baseline. The fleet surgeon 
at CINCUSNAVEUR is insistent that new technologies should not be implemented 
without a valid business case analysis prior to deployment. That is what occurred in this 
case, however, and it is now necessary to begin to study the effects of telemedicine at the 
local level without the benefit of a baseline. 

Although Rota received an ENT scope and a dermatology scope as part of the 
initial setup, these pieces of equipment were later shipped to satellite clinics while Rota 
retained the VTC and ordered new scopes as replacements. This was done in order to 
give more remote areas access to a medical treatment facility. The VTC was placed in 
the emergency room in order to provide access by medical personnel and also to 
emphasize ownership by healthcare providers. 

At a meeting held in Naples on June 4,1996, Rota had identified four areas as. 
consultation capabilities to offer surrounding sites: dermatology, ENT, urology, and 
radiology. Shortly after, it was concurred by representatives CINCUSNAVEUR, the 
MTF’s and NMIMC that a dermatology study would be initiated between Naples and 
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Rota. Sigonella, Naples, LaMadellena, Gaeta, and Souda Bay received derm scopes and 
the dermatologist at Rota prepared to receive consults. 

B. DATA COLLECTION MODEL 

It is proposed to use a four-area model to collect telemedicine data using surveys 
of health care providers, outcomes measurements, site profiles, and the utilization rate. 
Each area can affect the others. Additional information can be gleaned by examining 
them together. For example, if outcome measurements identify a positive change in how 
care is provided, the physicians would reflect that in a satisfaction survey and the usage 
of the tool should increase. The usage rate can then be displayed in the sites profile. 

Thus the relationships are described as causal and an inconsistency between any of the 
areas can identify why there is a positive/negative utilization. 

The following four-area model illustrates the flow and influence of data collection 
across the spectrum of health care. The model identifies two levels of relationships 
among the entities. A bold line represents a strong relationship with direct influence and 
the dotted line represents a more subtle but identifiable influence. The function of the 
surveys is to collect physician satisfaction and utilization data particular to telemedicine. 
The function of outcomes surveys is to collect changes in outcomes and number of 
medevacs avoided. The profile information is data displayed to support marketing, and 
utilization data is collected and displayed to provide a view of how much telemedicine is 
being used-the bottom line. 
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Survey 


Outcomes 



Figure 4. Proposed Data Collection Model—Influences 


1. Health Care Provider Surveys 

As a measure of physician satisfaction and intent to use, it is important to 
emphasize that before health care providers are quizzed on the many aspects of 
telemedicine it is vital to educate them on the subject to avoid ignorance and confusion. 
Understanding the questions will provide a much better answer from health care 
providers on an assessment than if they had never been exposed to the concept. The 
following questions were asked, and other questions follow that can be used. It is 
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especially beneficial to perform the assessment with a face-to-face interview, if possible. 
This has several advantages. First, it is not necessary to rely on the physician to complete 
the survey at their leisure and return it, which would most likely result in a poor response 
rate. Second, it is beneficial to place emphasis on the importance of establishing a 
personal cotmection with each of the interviewees, ensuring the interviews were 
interactive and collaborative in nature. This results in a building of trust among 
information systems managers and health care providers and also allows interpretation of 
non-verbal language. The focus of this survey is to determine what has worked and what 
has not worked for the provider in the implementation of telemedicine. This is especially 
interesting because it looks inside the heads of physicians. 

Demographics: age, years in the Navy, months at present location, job 
specialty, years in job specialty, rank. These are collected for comparison and possible 
trends. It is sometimes easier to understand responses if the assessor has a documented 
mental profile of the respondee, and is especially useful if a face-to-face interview is not 
possible. 

Has your use of telemedicine been successful? Why or why not? This is 
oriented toward the user who has been using telemedicine long enough to make a 
conclusion about the program’s success in general. Kotter (1995) identifies eight major 
errors made by organizations during a planned change: 

• Not establishing a great enough sense of urgency 

• Not creating a powerful enough guiding coalition 

• Lacking a vision 

• Undercommimicating the vision by a factor of ten 

• Not removing obstacles to the new vision 

• Not systematically planning for and creating short-term wins 
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• Declaring victory too soon 

• Not anchoring changes in the corporation’s culture 

This question attempts to identify reasons why the organization as a whole is having 
difficulty implementing a plaimed change. If these options are listed, the question 
encourages the respondee to give an honest, big picture, “bottom line” and then to 
provide further comments to support it. Of course there can be many reasons why 
changes fail and Kotter continues to state that even successful change efforts are “messy 
and full of surprises.” But a vision of the change process can reduce the error rate, and 
fewer errors can spell the difference between success and failure. 

Have there been positive/negative impacts on how you practice medicine (i.e. 
time required, quality assurance)? Describe. One of the problems with implementing 
a new process is the impression that there will be more to do or the current process will 
become more complex. This question attempts to find out if telemedicine is making 
healthcare easier or more difficult in the eyes of the healthcare provider. 

Do you feel you are proficient with information technology? Lack of 
experience or training has often been a barrier to implementation of any information 
system. A new piece of equipment is often met with the fear of the unknown and fear of 
change. Acknowledging this is half the battle, but once they connect learning a new 
discipline to solving an important problem, there is momentum to acclimate with the 
technology. In addition, interfaces are still being designed that are much better at 
bridging the jump from machine to user. A bad interface will cause adults to be “turned 
off’ and claim to be computer illiterate (Negroponte, 1996). 

Do you feel that there has been adequate management support at (1) your 
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site, (2) in the Department of the Navy, (3) In the Department of Defense, (4) in the 
private sector? This question is centered towards bridging the gap between clinical 
personnel and management across organizational levels. Often the burden of a 
unsuccessful implementation is shifted to an external entity and the fundamental problem 
is no longer in the local scope. This question attempts to determine if there is a 
deficiency with management support, and more importantly, where they perceive that 
deficiency to be. 

Do you feel that there has been adequate clinical support at (1) your site, (2) 
in the Department of the Navy, (3) In the Department of Defense, (4) in the private 
sector? This addresses the same issue as the previous question, but the subject is now 
within the healthcare provider’s- culture. 

Do you feel outcomes in the form of diagnoses, treatment, or dispositions of 
the patient have improved as a result of telemedicine use? Give an example. This 
provides a general answer to the question of whether or not there are changes in clinical 
outcomes. This can provide useful information in the case that a specific outcomes 
management protocol is not in place or under evaluation. It could also prove useful to 
compare it to specific outcomes data gathered through a clinical outcomes data input 
sheet for each telemedicine consult. 

Which types of telemedicine do you use? (with listing) The listing of 
capabilities changes very quickly and the health care provider may not be aware of what 
can be provided with telemedicine. So not only does this identify what is currently 
occurring at the site, it subtly keeps the clinician more up-to-date with technologies that 
could be used. 
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The following questions are those useful in cases where it is yet to be determined 
which applications to consider for telemedicine implementation. In a needs assessment 
done by Grigsby for the University of Colorado Hospital (O’Connor, 1997), attitude 
assessments are broken dovm into attitudes toward the hub facility and attitudes toward 
telemedicine. A primary determinant for a telemedicine program’s success is the 
willingness of the remote site to engage in a cooperative, collaborative relationship with 
the tertiary hub site. A lack of willingness may result from a variety of factors, but can be 
narrowed down to two general components: (1) ill feelings towards the tertiary hub site 
stemming from negative past experiences; and (2) the remote health care providers 
feeling threatened by the presence provided via telemedicine by the hub facility. The 
following questions are extremely open-ended and it is relatively necessary to conduct the 
assessment in a face-to-face meeting. 

What is your general impression of the hub facility? This is a question of how 
hi ghl y the hub facility is regarded by the local facility. The existing organizational 
dynamics may be negatively influencing the potential willingness to collaborate and 
engage in clinical dialogue. If there are preconceptions concerning the success of 
collaboration or a doubt concerning the level of hub expertise, the rewards for initiating a 
telemedicine consult may be preconceived to be non-existent. 

Do you have any negative attitudes towards the hub facility? If so, why? 

This question may bring out specific cases that have given the remote facility cause to 
avoid confrontation with the hub facility. Negative outcomes, whether they be business- 
oriented or human-oriented have a definite impact on willingness to collaborate. 

However, problems identified with this question are typically symptoms of a larger 
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problem not yet identified, or simply a misunderstanding that was never resolved. 

Do you have any positive attitudes toward the hub facility? If so, why? 
Again, this would bring out reasons to pursue telemedicine and identify critical success 
factors that can be capitalized by the remote sites, or even duplicated by other systems. 

Do you have concerns about how a telemedicine relationship with the hub 
facility may affect you or your practice? No system is ever completely free from 
potential problems and brainstorming the “what if s” could prevent negative encounters 
or provide scenarios for backup plans. 

What is your familiarity with telemedicine? As with the prior assessment form, 
this identifies whether or not the health care provider is aware of the capabilities and 
limitations of telemedicine. It is helpful to know if the healthcare provider has had 
hands-on training, or has done extensive scholarly research on the use of telemedicine. It 
may also help to identify potential clinical champions. 

Do you have any concerns about the use of telemedicine? This is the catch-all 
question regarding the technological or managerial issues that can change almost daily. 

What is your perception of the possible advantages and disadvantages of 
telemedicine? This is more of a “faith” question regarding telemedicine and may 
provide answers in response to the education received thus far. Health care providers will 
be determining whether the possible advantages become true advantages or evolve into 
disadvantages. 

Would telemedicine benefit your practice? Telemedicine is not suited for all 
specialties, i.e. those specialties requiring smell or touch and less cognitive in nature. 
However, there may be applications that have never been tried but have potential benefits. 
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What is the likelihood of you using telemedicine? Even if it has been identified 
that use of telemedicine should result in benefits, there may be underlying reasons why 
the physicians will not use it. If the physician is not yet ready for the change, it would be 
better serve the site to wait until obstacles have been removed before proceeding. 

How often would you use it? In anticipation for implementation, a baseline 
penetration rate provides a goal to achieve. If measured and monitored correctly, the 
target is a benchmark of success and a way to anticipate whether the application will be 
cost beneficial. It has also been found that participation increases perceived system 
quality, which in turn increases involvement. Involvement then influences both the 
attitudes towards the system and system usage. These attitudes together with the 
subjective norm are hypothesized to influence one’s intention to use the system (Bensabat 
& Moore, 1992). 

Would you actively or passively support the development of a 
telemedicine/distance learning program with other sites? Important to any new 
implementation is the role of the clinical champion. Taking the lead and becoming 
involved in their circle of peers can make or break a new implementation. If these 
champions can be identified early, momentum can be maintained and a healthy balance of 
experts will sustain the program. 

2. Changes in Outcomes 

The following questions were included in a form used during telemedicine 
consults with the purpose of collecting changes in outcomes and medevac information. It 
must be used consistently for a specified amount of time in order to provide an unbiased 
measure in which all consults are used and not just favored cases or environments. These 


47 



measvires can then be placed appropriately within the site profile to confirm the 
application at the site when it is felt that the application is acceptable to be used for 
resource sharing. It is also important that the health care provider realize that the 
measurement of outcomes is not concerned with physicians doing a “better” job. It is 
concerned with physicians being more certain of the outcome and having the tools to 
provide a more timely treatment. In addition to outcomes, medevac information has been 
embedded as questions to support measures of reductions in medevacs. 

Date, length of consult, patient FMP/SSN. The date is recorded for tracking 
purposes and as a way to uniquely identify the consult. The image, or other supporting 
digital information, is stored using a locally devised but uniform combination of the 
patient’s last name, family member prefix, social security number, and date as an 
identifying stamp. The length of consult can also be used to glean information on how 
long telemedicine consults are taking and to give the provider appropriate workload 
credit. The Clinical Health Care System {CHCS) requires a patient to be registered prior 
to receiving care. Since this is not achieved quickly and easily by the physician, the 
physician does not use CHCS and the manual record remains the easiest way to document 
that the provider is spending clinical time providing telemedicine consults. 

Permanent duty station, age, and service of patient. This is essential in 
providing patient demographics to be used in strategically and methodically locating 
health care providers. Since there may only be one specialist in the geographical area, it 
would be wise to locate him/her centrally—closest to where most of the patients are or 
where they are anticipated to be based on past trends. In addition, this information in 
conjunction with the service of the patient and whether they are active duty, can provide 
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data that may prove useful in performing a cost savings study. 

ICD-9-CM (International Classification of Diseases, Ninth Revision, Clinical 
Modification). This is a classification system that groups related disease entities and 
procedures for the reporting of statistical information. The clinical modification of the 
ICD-9 was developed by the National Center for Health Statistics for use in the United 
States. The use of ICD-9-CM promotes uniformity of health care data by establishing 
coding and classification standards and guidelines across health care settings. It is the 
core of the information infrastructure. The ICD-9-CM must be as accurate as possible in 
order to arrive at a binding clinical decision and be able to validate the continuum of care. 

Is the image acceptable to be used as a determining factor in your consult? 
This is an often overlooked question that plays an important part in a measure of success. 
If the image (or other digital object) is not of diagnostic quality according to the provider, 
it must still be known that the encounter occurred. Telemedicine consults that are not 
usable are opportunities for technical improvement. Only the specialist is qualified to 
make a diagnostic decision, and he/she is the one to be held responsible for the decision 
to use the consult. 

Image ID#. This should be used for definite tracking and can be used to verify 
that the correct digital object is associated with the patient’s record. 

Is a medevac recommended based on this consult? To where? This is directly 
related to a measure in medevac avoidance. In most cases, if a telemedicine consult 
cannot be performed, the medevac is initiated. If the telemedicine consult is performed 
and the physician is not comfortable making a diagnosis based on the image, it is most 
certainly appropriate to initiate a medevac. Another scenario exists where, based on the 
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telemedicine consult, the specialist wishes to continue care in person. The piece of 
information most useful to the medevac MOE is the number of times that a medevac is no 
longer required based on a telemedicine consult. It represents a direct savings of time and 
money and is a “bottom line” to provide as useful information to the local commander 
regarding the cost of running a practice, and also supports assessments and conclusions 
when compared to the trend of aggregate medevacs. If a medevac IS recommended it is 
also useful to know where the patient is being sent, again, to provide demographic data 
and support the appropriate location of specialists. 

Is a non-medical attendant requested? In approximately half of medevac cases 
overseas, a non-medical attendant is approved to accompany the patient to the consult 
site. This is mostly due to possible traveling difficulties due to the foreign customs, 
languages, and cultures that may prove uncomfortable to patients. Again, this provides 
information that can be used in cost savings studies. 

Recommended treatment and disposition. This is standard information for 
consults and documents the appropriateness of the health care provided. 

Based on this consult, has the diagnosis/treatment/disposition changed? This 
is the key piece of information useful to outcomes research. The specialist performing 
the consult alone knows if there could have been a different outcome without the use of 
telemedicine. Furthermore, each application is different and any combination of the three 
may have incurred a change. The specialist is also cognizant that the use of telemedicine 
is making a difference and the question may provide a positive reinforcement and further 
use. 
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3. Site Profiles 


The objective of this set of questions is to provide an up-to-date source of 
information in order to support the network necessary for medical collaboration. The 
Department of Defense medical network of providers spans the globe. Currently, in 
Europe, the sites have developed an informal, undocumented method to determine health 
C 2 ire needs in the area of response. However, there are many pieces of information that 
are difficult to obtain and often that information changes depending on who is asked. The 
Defense Medical Regulating Information System (DMRIS) is not providing an accurate 
comprehensive list of medical capabilities although it has the capability of doing so. 
Furthermore, the physicians do not necessarily have access to DMRIS; thus, when they 
need additional consultation, they find the outside source themselves and arrange for an 
accepting physician for a medevac. DMRIS has no requirement for accommodating the 
needs of telemedicine—it serves as a medical regulating function by facilitating requests 
for patient transportation. Patient regulation is what telemedicine is trying to avoid. 
Optimally, the system supporting telemedicine should provide quick and reliable access 
to a profile of each specialty and sites with telemedicine capabilities. 

Profiles can be developed and advertised to not only the area of response but to 
the rest of the Department of Defense for collaboration at any time and anywhere. As a 
result of the information gathered from the surveys and clinical outcomes, a profile can 
be generated to provide that information outside of the command for further exploitation. 
At a minimiiTn, the following should be part of a telemedicine profile: facility type, store 
and forward vs. VTC capabilities and standards compliance for specific specialties, 
physicians available for specific specialties, bandwidth available, and use of the WWW. 
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Facility type. This identifies whether the treatment facility was a clinic, hospital, 
or teaching hospital. It should also include a telephone number for general information. 

Store and forward vs. VTC capabilities and standards compliance for 
specific specialties. It is necessary to know whether the technology is compatible 
between the sending and receiving sites. Store and forward would require a compatible 
file attachment protocol like binhex, MIME, uudecoded, or FTP. In certain specialties 
like digital radiology, it is also necessary to know if the compression format is standard, 
such as with the DICOM3 standard. The VTC also must be compatible with standards 
such as H.320. 

Physicians available for specific specialties. As discussed before, specialized 
medical care cannot be provided at every installation, and it is necessary for the patient to 
have access to these specialists and for administrators to know where these specialties 
are. The telemedicine profile should identify those health care providers that are 
proficient with the use of telemedicine and it should identify a way to contact them for a 
consult. This can be expanded quite extensively. Some sites have developed a web- 
based physician profile for viewing which describes the physician’s practice. 

Bandwidth available. This question is necessary in order to anticipate the 
resolution of an interactive VTC consult. Sometimes all that is needed is for a physician 
to direct attention to a general area by pointing in order to emphasize the problem to a 
remote site. In this case, a 128KBPS may be adequate. In other cases, however, it mi ght 
be desirable to see an ultrasoimd with enough speed and resolution to diagnose and a T1 
connection, may be required. 

Use of the World Wide Web. Currently in Navy medicine, this is a very 
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inexpensive way to gather and disseminate information. As a result of a CHCS 
telecommunications requirement, Navy medical treatment facilities are all connected or 
have access to a global Wide Area Network (WAN) with a minimum rate of 56KBPS and 
most sites have constructed a home page. Pictures, the site profiles, a feedback page, and 
even an electronic consult can easily be constructed and viewed by any computer 
connected to the network. Part of the profile would be a “home page” to go to for 
information on the site’s telemedicine capabilities, objectives, measures of effectiveness, 
and results of surveys. The barrier to accomplishing this is simply lack of education 
and/or initial staffing. 

4. Utilization Rate 

Currently this is being gathered manually by physicians providing the 
telemedicine consult. Since consults are still few in numbers, it is relatively easy to 
document how many consults have occurred and how long they took to complete. Within 
CHCS, there is an option to track “teleconsults.” This was originally designed with 
telephone consults from patients in mind and not for tracking or transferring images and 
consulting by VTC. Kno'wing how much the technology is being used is important since 
it is the bottom line for any information system. It may be the most efficient, elegant 
system in the developer’s mind, yet it is not being used. This serves to identify that there 
is a fault occurring in the overall system and all prior efforts will be for naught imless the 
fault is identified and corrected. Utili 2 ation is the most important and revealing measure 
of telemedicine. It has been suggested that the real benefit to be realized by telemedicine 
is that of increased use (Edwards, 1995), which he calls a “second order benefit” and is 
the goal of any new system. 
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C. APPLICATION TO THE DATA COLLECTION MODEL 

Using the data collection model previously described, a case study is presented 
using the environment at the Naval Hospital Rota. 

1. Health Care Provider Surveys 

Two days prior to my site visit to Rota, a short survey was distributed to 
healthcare providers and information systems personnel. Upon arrival, I was greeted 
throughout the chain of command and given an opportunity to speak at several of the 
clinical meetings held in the command. During these meetings I informed listeners that I 
was there to gather information across all functionalities to include attitudes and to 
identify where telemedicine was used or desired. I was also granted time to speak to 
several physicians, nurses, and hospital corpsmen. I distributed the same survey to Naval 
Hospitals Sigonella and Naples remotely as pilot surveys; however the response rate was 
significantly lower at around 34% and 17%, respectively. I attributed this to lack of 
visibility and, from what I was told, an overwhelming number of surveys being 
distributed within the last year. Furthermore, the previous surveys had not reported 
results back to the command and there was not an incentive to provide survey information 
based on the merits of the survey alone. 

There are 36 doctors and dentists stationed at Rota, and 23 responded to the 
survey for a response rate of 64%. When comparing this rate to that at Sigonella and 
Naples, it is easy to see how visibility and top-level support can make a difference in 
participation. The statistical data, which may prove useful in later studies, resulted in the 
following; 


54 




• The average age of the respondee: 40. 

• The average time served in the Navy: 15 years. 

• The average time at Rota: 16 months. 

• The average years in specialty: 9 years. 

The results of the survey are as shown in the following table: 


Has your use of telemedicine been successful? 

8 yes 

4 no 

Have there been positive/negative impacts on how you practice 
medicine? 

8 positive 

1 negative 

Do you feel you are proficient with information technology? 

14 yes 

9 no 

Do you feel that there has been adequate management support 
at your site? 

16 yes 

2 no 

in DoN? 

11 yes 

5 no 

in DoD? 

9 yes 

5 no 

in the private sector? 

7 yes 

2 no 

Do you feel there has been adequate clinical support at your 

15 yes 

3 no 

site? 



in DoN? 

16 yes 

3 no 

in DoD? 

9 yes 

3 no 

in the private sector? 

6 yes 

2 no 

Do you feel outcomes in the form of diagnoses, treatment, or 
dispositions of the patient have improved as a result of 
telemedicine use? 

11 yes 

1 no 


There was additional space for each question to provide further explanation and rationale 
for the responses. This proved especially helpful in interpreting non-responses and 
negative responses for specific questions. For example, the radiologist stated that he did 
not believe telemedicine was successful at Rota because they did not have the appropriate 
equipment for true digital radiology. It is also noteworthy that respondees mostly felt that 
Rota was doing better than other sites and civilian counterparts. Although valid 
conclusions about Rota cannot be made based on the small number of respondees, the 
general feeling is positive in terms of what has been done locally and generally with 
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telemedicine. 


2. Changes in Outcomes 

Two doctors assisted in constructing a data collection form to be used for 
identifying changes in outcomes for telemedicine consults. From the period of Nov 96 
through Jul 97, using this form, the dermatologist, identified the following: 

• There were 7 changes in diagnosis. Upon his observation of the image the 
initial diagnosis of the remote consultation changed. 

• There were 9 changes in treatment. The initial treatment changed, possibly 
related to the change in diagnosis. 

• There were 11 changes in disposition. Again, this change in the patients status 
makes a direct impact on the readiness of the active duty member. 

• Of the 14 consults, 12 were considered acceptable images for a consultation. 
The other two were considered a poor picture, or of inadequate resolution. 

• There were 9 consults identified as “non-medevacs”. In other words, if there 
had not been a telemedicine consult, these cases would have been medevaced for 
observation elsewhere. 

Again, although statistically binding conclusions caimot be made, there is an 
implication of favorable outcomes and a definite avoidance of medevacs. This provides 
the site with a basis for evaluating telemedicine as being effective. 

3. Site Profiles 

Currently, the network of outside consultation within the DoD network is 
accomplished by informal, imdocumented methods. In short, there is not an acceptable 
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standardized system of providing a visible profile usable for the fimction of telemedicine. 
At a minimum, needed information is facility type, store and forward or VTC capabilities 
and standards compliance for specific specialties, physicians available for specific 
specialties, bandwidth available, and use of the WWW. For example, the Rota profile at 
a minimum, is found to be: 

Rota: Naval Hospital 011-3456-82-3673, DSN 727-3673 
S&F tmedrota@med.navy.mil 
VTC 011-3456-82-5555, PictureTel, H.320 
Dermatology - Dr. Pidkowicz 011-3456-82-5555 
ISDN - 256KBPS 
http://192.101.124.40/ 

4. Utilization Rate 

The utilization rate reflects the level of use in comparison ’svith traditional 
methods. At Rota, the number of telemedicine consults were almost entirely attributed to 
dermatology. During the observation period (November 96 to Jul 97), there were 14 
dermatology consults. There were approximately 400 dermatology consults during the 
same time period for a aggregated utilization rate of 3.5%. This leads to questions 
regarding possible cases that were not presented for use by telemedicine. For a short 
time, one of the satellite hubs contracted the services of a local dermatologist for cases 
that could have been serviced by telemedicine. This leads to an assumption that there is 
still sufficient skepticism by many healthcare providers. There has been minimal 
education which ultimately will negatively affect use of any system. Since there have 
been no negative experiences with telemedicine noted or comparative data (the Rota 
dermatologist is the sole Navy provider in Europe), the utilization rate can be 
benchmarked as an introductory measure and monitored for change. The only other 
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consults were 2 pediatric VTC consultations, and other specialties such as radiology have 
yet to be explored for use, with the understanding that the specialties must not be 
aggregated into one because of the variant suitability of telemedicine to different 
functionalities. Regardless of the result, it is necessary to document utilization rates and 
observe trends in order to identify potential investment areas. 

D. GAP ANALYSIS: NEEDS ASSESSMENT 

A needs assessment addresses the attitudes of the physicians and this is where a 
physician survey can prove invaluable. Additional questions can be framed to obtain 
information regarding the skills and interests of the health care providers, how busy their 
practices were, and to gather their opinions on the health care needs of the community. 
Consequently, this involves a more comprehensive and subjective survey than the one 
distributed to the physicians at Rota. A general needs assessment was not done at Rota, 
thus doing a satisfaction survey may create confusion. This is illustrated by eight surveys 
that were entirely blank except for a comments that it was not used or they didn't know 
what it was. The same number of surveys were complete and descriptive, with the 
remaining surveys being complete and brief. This illustrates an inconsistent level of 
knowledge among users at Rota. This is a gap in the system identified with the use of the 
surveys and this lack of education can be considered to be a contributing factor to the 
low penetration rate. 

When it is rationalized that it is rmethical to consider costs and thought of 
qualitatively, telemedicine is always considered appropriate if it MIGHT have benefit. 
Using the utilization rate can support requirements for telemedicine. Although health 
care providers at Rota believe telemedicine has benefit and outcomes are favorable (i.e. 
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consults are identifying favorable changes in outcomes), they are using telemedicine at 
what appears to be a low rate based on twice the rate of dermatology consults that 
occurred in Bosnia. This also suggest a resistance to the change in business practices, 
and that significant training, education, and dialogue needs to occur before telemedicine 
can provide optimal benefits. 

It has been recognized (O’Connor, 1996) (Allen, 1996) that telemedicine 
technology hype and its potential to cure some of what ails the health care system results 
in a scramble to implement telemedicine networks. This rush is at the expense of a 
rational, objective process to determine whether a telemedicine program is actually 
needed, let alone welcome. After administration and analysis of a needs assessment, a 
strategic plan can be developed outlining the needed services as well as the most 
appropriate telemedicine technology to provide those services. A needs assessment after 
implementation should not answer the question, “Did we do the right thing?” It should 
answer the question, “What should we do?” 

In summary, looking at the qualitative measures, physician satisfaction is 
favorable, outcomes are favorable, medevacs have been avoided, but utilization is low 
which suggests a resistance to change. Telemedicine should be integrated into the 
tr ainin g and education of all personnel and its success should be marketed outside the 
command through the use of the World Wide Web. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

Medical decisions are certainly complex, highly variable, and poorly supported • 
with clinical studies. This is a difficult foundation on which to place a telemedicine 
program. But there is a push by DoD physicians and managers to be a testbed and 
identify successes and failures. This data is essential to prove the effectiveness of 
telemedicine. The methodology shown in this thesis is a proposed beginning to establish 
and Tnaintain a system for measuring the effectiveness of telemedicine. Measurements of 
effectiveness (physician satisfaction, outcome measures, medevacs avoided, and 
utilization) can support telemedicine. Based on evaluation of these measures, a gap 
analysis can identify where resources should be directed. 

B. RECOMMENDATIONS 

While traditional consults are documented in the organic Composite Health Care 
System (CHCS), a telemedicine consult is not, since the patient is not registered outside 
the organic CHCS. Thus, methods for gathering data remains a manual process, i.e. 
tedious and error prone. The methodology stated here is dependent upon the commitment 
of everyone in the process to contribute to data collection. It would increase the 
reliability of outcomes and utilization data to ensure that all telemedicine consults are 
captured by CHCS for further use by planners. 

It is the responsibility of each local facility to maintain an appropriate set of 
measures to support locally defined objectives. Each facility has a unique strategic 
mission leading todifferent objectives to be achieved. Furthermore, measurement is an 
iterative process, requiring successive refinement from one year to the next and a 
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program must be in place to ensure this. It is important that the program be driven from 
the top down and top officials should understand the weaknesses in measiures of 
effectiveness, and use them strictly as indicators rather than absolutes. They should be 
kept simple so that they are imderstood, meaningful, and useful. 

It is a mistake to think that further deployment of new technology will succeed 
without the involvement or "buy-in" of the health care providers. Taking the time to do 
face-to-face interviews and engaging in productive dialogue between managers and 
healthcare providers can produce much more information than by second-guessing and 
filling out forms. 

Needs assessments must be timely and iterative. Ex post facto studies have little 
value in a planned change where it is necessary for users to be actively involved in the 
change while it is happening, not after. Needs assessments should be focused on the 
strategic goals and objectives before implementation of the technology. 

C. AREAS FOR FURTHER STUDY 

Outcomes management is an area that merits further study, especially when 
observing the various changes that may occur due to a telemedicine consultation; 
specifically, changes in diagnosis, treatment, and disposition. The interaction of one 
change to another is a complex situation that may prove to be of immense value in the 
study of outcomes. A clinical data set with an adequate number of data elements must be 
collected and categorized by specialty to support this. Outcomes management is of 
increasing interest with managed care specialists, and the changes in types of care are 
especially of interest when examining impact on the healthcare system. 

Further effort should be spent on a more in-depth analysis of the validity of the 
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proposed MOE's, possibly through the use of a Delphi group. 

Additional research is also necessary in the development of a useful needs 
assessment to include physician surveys as new telemedicine methods become available. 
This should focus on creating the right mix of questions to be asked in order to provide 
the most useful feedback. 
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